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The construct validityiof scholarly productivity was 



ABSTRACT 

The construct validityio: 
investigated, with attention to definitions and measurement 
approaches. A nonrepresentative sample of 51- schools of education was 
selected to include very producjtive schools. Measures were made of 
the following variables: citations, rankings froAi previous studies, 
publications, conference participation, fundiirg, and general 
institutional chara<^teristics . Publicati^on counts were estimated ^rom 
data in tl^e Educational Resources Information Center (ERIC) system. 
Each school of educ|ition's participation in the American Educational 
Research Association conference. was assessed for 1981 and 1982. 
Information on grants awarded to schools of education during 
1978~ir9j32 was obtained from the SmithS9nian Science Information 
Exchange. Finally, the higher Education General Information Survey 
provided data on five variables: percentage of doctorates granted, 
total research expenditures, total overall expenditures, government 
grants and contracts per full~time equivalancy, andj. Average fi^culty 
salary. Conclusions include the following: ^program size did nor\ 
significantly determine prestige; surveys on reputation did not 
appear to be reliable; and citation counts seemed to be^ the best 
measure of scholarly work. (SW) j . / 
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# Iri. assessments of ^ excel 1 ence in higher education, (questions 



and 

\ 



of scholarly productivity are often pivotal.' Research and 
publication are the driving forces in the modern university 
eva-luators of university programs are attuned ^o tbis fact. 
From i rfd i vi'duar promot i on and tenure decisions to overall 
i nsjt'l tut i onal prestige, scholarly research is a flindamentJl * 

i ssue. ' " ' ' « 

A variety- of procedures h^ve been used to measure 

productivity. Some researchers have surveyed faculty or ' . ' 

admi ni strator^ to obtain ranjtings of universities and 

department,s. ^Ot^hers have used the- quantity or quality of 

publications to rate programs, sometimes including present^ti^9ns 
* .1 

of papers at conferences. Another measure is the type and 
amount of funding obtained by a department, and, at <th<p 
university level, characteristics such as faculty salaries and \ 
research expenditures have been Vinked to faculty producti y itj( . . 
FiUaJly, citation rates, a measure of peer recognitjon, have 



been used to* assess scholarly accomplishment. In some way , -e 
of thesp measures has been associ at ed ,wi th product i^vity . 

. This study addressed th,e construct val idit^; of • Scholarly 
productivity: H^ow this concept has ."been defined, and how 
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various ways of measuring it compared. In particular, data 

♦ • • .1 

pertaining to schools of education were examined. The • ^ 
perspective was-that of an administrator or evaluator who must 
collect and assess information regard'ing productivity, 
discovering in the^rocess what data are available for making 
.valid, useful comparisons among sctioojs of educaticin» what 
problems occur in gathering this data, what relationships exist 
amoijg measures, and what interpretations can be made. These 
issues are of considerable practical significance, reflecting 
processes that often have a direct impact q\ t>ie future of a 
program, a school or a career', ' 
Interrelationships among va ri ou s^measu res of product i-vi ty 
have been explored in different ways by several researchers. 

Hagstrom (1971) determined that a single, unrotate^d factor , was 

■ ■ ^ ' . 

sufficient to account for most of the variance among 188 

uni ver^i ty^ science ^and mathematics departments on ten variables 

related to institutional quality. Astin and Solmon discovered 

< 

J 

two f actors, ""'scholarly excellence of faculty" and, "commitment 
to teachi ng ,". in their factor analysis of a questionnaire on 
undergraduate excellenqe (Astin & Sol mon , 1 981 ; Solmon & Astin, 
1981)." They- also found evidence that oyerall institutional • 
prestige is an important aspect of individual departmental 

# * ' * - 

ratings. Another study ^(Andersen. Narjn. « McAllister, 1978) 

used regression techniques to predict prestige rankings from 

various publication and citation data, finding that variables^. 

related to size, as well as to quality', are predictive of 
.. ' *' ' . J ' ' 

pre^^tige. Endler, Rushton and Roediger (1975) c^orrelated both 



puJ)l i C4.t i on counts and 'citation data with other measu res , 
concludinq that ci tat i on rates • may be better than publication' 
counts as a\neasure of quality, . Eash (1983) produced composite 
rankings of schools of education using conference presentations 
and publication counts. ^ . • ' . ^ 

Thi s ^tudy ^ exami ned rel a t i onshi ps among a series of 
•jTiedsuipes of productivity for a single sample of schools using 
colnputeri zed data bases easily accessible, to any researcher. In 
particular, the results of citation an'alysis, a method not often 
used in many fields, including educat i on 4^ roc , 1984)* was com- 
pared with other datj.^ " ^ 

r 

* . \ " 

* ^' Method ^ 

Sample ^ • 

A non- rep resentat i ve sample of 51 s.chbqls of education was 
chosen from rankings in other studies t.\d include as many of the 
most pVoductive schools. as possible^ (see Table 1), For each^ 
school, faculty lists were obtaine(J^ from 1981 course catalogs or 

•by writing dt.rectly to the university. Measures v<ere made of 

I . , 

the following variables:' citations, rankings from previous 



studies, publications, conference participation, funding, and 




general institutional characteristics. .Datan'or the 4600 « 

faculty vjnclyded in the study were- aggregated at the school / 
1 e V e 1 • J . * V 
C i tat i ons ^ . . . 

• ■ ' ' • 

The number of citations for each faculty member in the 1981 
Social Science Citation Index (SSCI) Was counted. The SSCI . 
ists, by author, citations of that author ' s. work du r1 rig a given 



year. A .set of rules w«s developed to standardize the counting, 
proqedure, and two judges were trained, enabling an assessfnent 

to \)e made of the reliabrility of citation counts^. Th*e obtained 

\ f' ' * • * . ' • ' 

1 ntejjrat'er-»^rrel at i on coefficient of .r««.90 ipdicated that. 

citations can be re.Uably, counted. 

' • ^' . ' . ■ ' ■ 

^ ^ The,^^5CI pro.ved to b^e a, 'rich source 'o/" data.-fylly. ♦ 

j^ef^renci ng' 160 education journals, as *weT\ as|inajiy other 

journals in rejated f.ields, on an annual basis. A more thorough 

discussion of SSCI citation analysis occurs elsewhere (Kroc, 

1984). . ' . . . . 

Rankings from Previous Studies 

^ S , ^ ^, 

1 

.Rankings of schools of education from nine studies were 
recorded. Five of these research "efforts (Blau & Margulies, 
1975 ; Cartter, .1-977; Ladd & -Li pset , 1979 ; Sieber. 1966; 
Wa 1 berg 1972 ) were surveys of, either education faculty, A£RA 
members, or deans of schools ^of education. The other four 
Studies prodcued ranks based on publication counts (West . 1978) , 

AERA sDresentat i ons (Dole, 1981; Schubert, 1979) or both (Hash, 

' ' " > . I 

1983). * • * . ;. ^ 

Publications ^ v ^ 

Two data bases. Research in Educ^ation (RIE) and the Current 

•Index to Journals in Education (CIJE) were searched using the 

ERIC system. The high cost of obtai ni ng -thi s information 'for 

each facul ty • member ensured that there were limitations on the 

amount of data, that could be collected and on the number of 

faculty surveyed, resulting in a decision' to sample 25% of' the 

4600 faculty. Based on previpusly obtained citation data. 



faculty from each university were. split Ijjto four strata, and an 
optimal allocation 'formula (*Coy(s^ran 1 953 , p. 93) was used to 
determine the proportion of otjservat i ons within each strata 
which wbuld minirtiize the standard error, of the mean publication 

'>counts, • . . . 

. , ./ 

C 0 h f e r e n c e , P a r t i c i p a t i on - . , 

' Conferences participation was measured by counting the ' 

number of presentations listed in the . annual meeting programs 

, . ■ c 

at two conf erencesjw-t he American Educational Research Associa- 



tion (AERA) conference in 1981 and 1982. Two consecu t i ve AERA 

conferences were surveyed to ^control for bias due ^o the ineeting 

^ocation; The 1981 conference was h^ld in Lps'Anqeles, while 

the 1982 location was Boston, \ All types of AERA presentations 

were given equal weights, except discussants, who were not 
♦ 

counted. 
Fundi n g 

Only one accessible source of funding data was found, the 
Smithsonian Science Information Exchange (SSIE). an agency that 
compilecf and stored for computer access information on grants 
i ssu'ed between 1978 and 1982-. SSIE received project 
descriptions frdm over 1,300 organizations: federal, state and 
local government agencies; nonprofit asso.ci at i ons and founda- 
tionsj colleges and Hni vers i t i es ; and, 'to a 1 itni.ted extent , * 
private industry and foreign research organizations, Approxi-' 
mately 90% of the info/^mation was provided by agencies of the 
federal government. Unfortunately, .the SSIE was phased out late 
in, 1981, a victim of federal budget cuts. However, the National 



TechnicdV\Inf^rmation Servi ce ' (NT15) Ms assumed aj,p,t)rt ion of 
SSIE's former services, ^[jii does not yet gather djta from the 
agencies whicih are mbst'^Uke.ly to fund educational research. 
The dissolutjon of the. SSIE was not a problem in this study, 
since funding data was* available through 198K , 

The' organizat^ion of the SSIE data. base ma^ it ppssible to,, 
fonter the prograiif and university name at a computer terminal and 
receive a printout containing abstracts of the ^overnmeht* grants 
obtained by that education program between 1978 and 1982. Each 
abstract listed the grant ' s » invest i gators , title, sponsoring , 
organization, performing organization, funding period, and . ^ 
dol 1 ar amount . , 4Jnf ortunately , thisMnformation was not always 
complete; in f att , 30% of the abstracts did not list a dollar 
amount for the grant. Still, it was possible to cou-nt the 
number of grants obtained by each program, ,as well as to get 
some indication of the money invo,lved. 
Institutional Cha racte r i st i cs 

All universities thatTeceive federal funding must "report 

/ ■ ■ . . 

certain information annually to the government.' This data is 

F 

documented in the H i gher 'Educat i on General Information Survey 
(HEGIS) and is, available on magnetic tape. Variables which were 
accessible and of interest for this study were dollar amount of 
government grants and contracts per PTE (1978-79), total 
research expenditures ( 19 79-80 ) i tota 1 expenditures (197^-80). 
percentage of doctorates granted (1979-80). and average faculty 
salaries (1980-81). Al 1 of these measures were at the univer- 
sityilevel, not at the school or department level. 



Analysi s ' . 

Two sets of rankings of schools of education were produced 
for each variable, one based on total counts for each school ^apd 
the other on total counts, divided by the number of faculty in 

* * . 

each' sch^Ql, Comparisons of }:he sjx sets of variables, using 
both unitTs of analysis* were m^de by examining correlation 
coef f i c*ients. The patterns of these coeffic^'^ts provided data 
on how the variables in this study were i nter-re| ated as well 
as how th^y compared with rtieasures of productivity froni^' other 
studi es % ' ' % 

. ' RESULTS , ; 

Rel at i ojishi p B etween. Ci tat i on Co unts . / 

and Ranking s from Oth er Studies . ' ~ 

' /' ~ ' ■'• H 

As discussed, a number of studies Of education schools, have 

; ■ ' ' . ■ .. ^ ' ■ * ^ 

been done. The ranking* resulting from these studies were 
correlated witxh on^ another as well as with citation d^a , 
producing the coefficients 4n^ Table 2. On the diagonals are the 
number of schools which were ranked in the study that is listed 
at the- top of the column. For example^| the ranks for ,23 
programs which were available from the Dole (1981) study, 
correlated .26 with rankings from the Ladd and Lipset (1979) 
study. Five sets of rankings (Cartter, 1977; Ladd & Lipset, 
1979; Sieber, 1966; BJau and Margulies. 1975; and Malberg, 1962) 
were based on surveys which asked respondents to list the top 

* ♦ ■ 

schools, of education. The correlat^i^ons among these: five studies 
showed consideriable variation, indicating how survey r.esponses 



can differ. For examp^^. both.Cartt^r ( 1977) and Ladd dflcl. 

iipset (1,979) surveyed eVcatKon-JacuT^ rnembers In ,at; tempts to 

discover the most disti ngu\shed sch'ools of educat i on, . Yet thei r 

rankings correlated only A^. Prest i ge 'rank,i ngs [ then . may not 

be Very consistent, even whea similar questions are asked 'ovf the 

same grbup of respondents. > ' , 

Studies that obtained less subjective measures of equality 

fared somewhat better when compared with one~ another. Schubert. 

(1979). Dole (1981) and Hash (1983) used AERA *p resentat i ons as 

measures for ranking programs. Schube^rt <;,ouTit«d total presea- 

tations. Dole us,ed presentations per faculty, and Eash reported 

both; The .ranks of D^ole a,nd E'ash show a correlation of .70, 

wl^il.e those of Schubert, and Eash correlate .^83, an indication ^ 

that AERA presentation counts are somewhat consjl stent 

^Similarly. Eas-h (1983) and W^st^(1978) ra^iked grograms ^ 

according ^o the |otal number of articles appearing 

. . « ' 

•education journals. The correlation between these 

r = .49, would probably have been tiigh/r had more of the same 

journals b'een surveyed. Nonetheless, ^here' is evidence .of 

♦ 

moderate reliability in ranking by counts of journal articles. 

Reisults of studies using total faculty counts showed low 
correlations with those calculating results op a p6r faculty . 
basis. For example, the AERA presentation ranks of Dole and 
Schubert correlate only .19. In general, ratings using the 
faculty member as the unit' of analysis correlate more highly 
with prestige rankings', as shown by comparing Walberg's (1972) 
ra/nkings with those of Eash. When asked, to rank schoo,ls:of • 




^adi ng 
studies , 



educationv su rvey ^ respondents seem^ to think more in terms of 

, research product.ivity per faculty member -th-an they do of total 

, ■ • • , ■ .• ■ ■ - • ■. • • • ■ . 

productivity. \' 

Mean" Gi tati on' ranks correlated moderately to strongly 'With 

■ ■ ■ . . " ■ - .^ 

all survey rankings exce.pt Ladd and Lipset's and. with 4-II other 

rdnkihgs which used individual faculty as the unit«of analys-is. 

The. highest correlations were with the Cartt'er survey or ■ 

.83) and with Eash's rankings of AERA presentations per PTE 

n(r - .65)^ Mean citation rank, then, showed a relationship . 

with the rank i nq^ methods used in ottier studies, ^ 

*• . ■ < . » 

Although mean citation rank correlated well with' other 

types of ratings, total citation counts were not as strongly-, 

related' to other measures. Percentage of faculty with ^^ero 

citations, on the other hand, followed a pattern identical to 

that of the fij^an citation rate, indicating that they both may be 

tapping the samie dime'nsion of productivity. ' 

Publ i cat ion '-^Counts , . \ 

■^ p i. . . - ,.^,,*. -w - ^i -- .1^. I. m ill... , ^ • 

I 

Publications were estimated from.data in the ERIC system. 

Two oth^r studies (We$t, 1978; Eash, 1983) used publication 

counts as a method for ranking programs. Table 3 shows inter- 

corre 1 atci ons^. among the ranks produced from thes'e various methods 

of measuring publications, while Table 4 displays correlations 

between publication ranks, citation measures , .and rankings from. 

other St udi es . „ . • 

• ' y ' . ' 

The consistency between Eash ' s counts of articles in the 14 

journals a-hd the publication measures in this study was shown by 

the strong relationship betwe'en Eash'.s total articles and the 



.total ERIC publication rank (r ^ .75) as well as by the 
correlation between' the two FTE counts (r .63)^ West's 
method of ranking (bdsed on total publication counts in 2. 
journjals) did not correlate as well^with either of the other 
measures (r « .33 wi th t ota 1 ,ER IC publication rank, and r * 
.49. with Hash's total counts). 

• The correlations between publication rankings and other 

' * I * 

ty().es of rarfkings showed a wide range of values. In general, 
though, the five survey results had more in common with per 
faculty measures of publications than with total counts^ four 
aut of^the five studies had higher correlation's with per faculty 
publication rates. ^ 

Citation ranks were not strongly related to publication 
counts, 'as shown by the coefficients in the first three columr\s 
of Table 4. Mean citation rate correlated only ,34 with gash's 
FTE publ i cation 'ranks and .39 with per faculty ERIC publi^cation 
ranks, while total citation rank showed a similar degree of 
relationship with the three measures of total publications. A 
stronger correlation, though, was found between the* percentage 
of faculty with no citations Jind per faculty publication rankS4 
It may be that tKose faculty who are more often cited have quite 
variable publication patterns, thXiS loweri ng any correl ation 
between citation and' publication rates, while tho^e who are not 
cited tend hbmogenously not to publish, thereby increasing the^ 
relationship between publication rates and the percentage with 



no ci tat'i pns . 

It should be note^l that the publ icat i on^counts in this 



study were biased on combining entries from both the CUE andy 

RIE. These publ i ca't i on caunts, t^eh , include b:^th journal 

articles and "fugitive" literature such as technical reports and 
/ ■■ ■ , , - . , • 

evaluations. ^ 

Conference Participation y - , " * " 

E*ach school of Education's participation in the American 
Ed-ucational Research Associ^t;ion' conference was assessed for two 
years, 1981 and 1982. , fhr.ee other studies used AERA participa- * 
tion as a means for ranKiV<g progra^ms.' Scjhubert (1979) used data 
from 1975 to 1979 to obtain countj^ of total presentations by * 
each school of education., Dole (1981) used Schubert's figures, 
di v*i di ng ,each total by the number of faculty to produce ran4i|igs 
with faculty as the unit of analysis. Eash (1983) counted AERA 
presentations over a s4ven-year period, 1975-1-981, comDei|feing 
both total and PTE rankings of schpols. Table 5 shows Tne 
i ntercorrelat i ons among these three studies and with fehe ranks 
from this^study. These coefficients indicatTed greater consis- 
tency among AERA studies than among studies of publicatiqn 

rates. 'This seems reasonably, given the ease withwhich AERA ^ 

C ■ 

participation can fee judged, from the program meeting notes, and 
the' fact that each publication study used counts from a differ- 
ent set of journals. - \^ , - 

. Table 6 indicates correlations between the various AERA 
'measures and three groups of rankings, representing citation 
rates, publication count^ an.d prestige surveys. Citation ' 
measures correlated better with AERA presentations than with, 
publication rates. Perhaps authors who are cited-more often 



have' d greater inclination to sut?mj t their wo rk't;^o; the nati bnal 
exposure off-^red by AcRA conferences.^ , •.. 
, Correlations between 'AE^i-A data ..and, publication rdnks showed 
a. pattern of moderate rel a ti onshi ps . vfli il e prestige surveys did 
not copr-elate consistently with AERA results, as rti qht , be of •. ; ' 
expec^ted . . ' ' . . ; 

Fund i ^ , r *. . ^ V' ' \ 

A computer search of the Smithsonian Science Information 
Exch-ariqe (SSIE) elicited lists of abstracts of grants awarded to 
schools of education between 1978 and 1982, From this informa- 
tion, the number of grants for each education program xis well as 
the dolj.ar amount was extracted, 

^ Correlations between funding data and other indices dre ^ 

. * * ■ * ' ■ ' . " 

',i/id'i c^tJeff i n Table 7* Hiqher cor rel a^t i ons are apparent for the 

^ ' ^ ^ ' . -rf"*- ' ' »- -'^^ — 

nu^mbef o^^^ (first two rows of the. table) as opj)osed to 

4al lar 'amoun[ts/:('l ast two rows), ^s mentioned previously, there 
was/d l,f f i'.cu.l ty- i n obtaining complete information on the amount- 

■.' ■ ■ ■ ■■ '■ "'v' >,■■'■' - ■ . ' . ■ • ■ ■ ■ . ■ ■ 

of - :•• t h-e--. .-ja ward •su^*'-/^^ // > ■ . ' / '• . . . ■ ' 

CI early ;;;x€Ht'a.ti on rates correlate better than publication 
c;b:unts or 'A-E:R.'A::^^ with funding data. If the casje is 

made^'t>>'at more pVoductive facultyaiSQ better ab.l e to obtain^ 
grants , t hen ^these ^-o^/re.l a t i ons. enhance 'the validnty of ' 
citation' counts -as an i ndeJc of • 'scho 1 a rly work^, particularly in 

■ ry..^}:/.!^ ...... , - . ' • . ■ ■ ■• . 

cofflpar i s.O';n^,^'.W^^ and AERA conference partici- 

pat jcn,. (t, jTi^y^^ 1^^^^^ more ci.ted scholars write stronger grant 

proposals^, or -tKiit tlieSr rep-utations influence the dtecisions 
pf the grant;^ing,yag^n<:ies. . ^ • « 
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Insti t ujiohal Characteri st i cs . ♦ 

' • / ^ ) , ■ - 

The Higher Education General Information Survey (HEGIS) w^s 

•i ' • . . . . », 

,th% ..source for data on five variables: percentage of doctprates 
grant^ed. total research expendi'tures\ total overall expendi- 
tures» government gra^its and contracts per FTE, and average, 
fiaculty salary, , A1 1 , of these, vari abl es were measured fo/ the 
entire uni varsity not at the school of education level, and 
reflect 'either the 1978-1979 or the 1979-1980 academic year,./* 
Table 8 shows correlations between institutional charac- 
teristics and four s.ets qf variables consldened in th^i? study: 
citation rates, publication data. AERA presentation ranks and 
funding information. These, coef fi c i ents show that data reflec- 
tive of the university as a whole was related to department- 

« 

level indicators of prodyct i vi ty . This was, tHbe for the schools 
of education iji this study across all measures. Perhaps 
universities which command greater resources, and hence can 
afford larger salaries and expenditures, are able to attract 
mor.e productive faculty. 

^s with the SSI.E funding data, institutional variables, / 
especi al 1y -^xpehdi tures , grants and contra(rts, and salaries, 
tended td correlate best with citation rates. Although the halo 
effect may have been an issue with these data, this does not 
explain" why citation rates are the strongest correlates. A more 
likely explanation may be that these institutional variables hjgi 
a direct or indirect influence on departmental productivity, and 
the number of citations were a better reflection of this. ^ 
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* Conclusion • 



* 

Few would clOMbt the importance of scholarly productivity in 
university, life; yet obtaininq measures suitable tor makinq * 
comparisons amonq "un i versj t i es rn^y be difficult ,and xontrover- 
sial. This, study, provided comprehensive data on several 
relevant variables for ^ selected sample of schools of educa- 
tton. Relationships amonq these measures 'suqqest several points 
related to construct validity* 

K — 

Fjrst, the tendency for prestige survey results to ^ 
correlate more hiqhly with means^ .rathe^r than with totals on 
other measures was an indication that the program size was not a 
si'qnificant determinant of prestiqe; quality was Hnore. important 
than quantity. . « 

Second, surveys had an^'annoying Cenderfcy to show inconsis- 
tent relationships with gine another. Although the small number 
oftschools ranked in some studies, as well as differences in 
when the assessments were made, may have contributed to t4i i s 
•problem, the re|,iability of surveys on reputati.on was^ot 
evident in these data. , ' . 

Thi-rd, Uhe moderate c*orrel at i ons found ^0"9 the various 
measures i ndi cated . cons i derab shared varia'nce and, perhaps, 
"^justification for thinkinq of scholarly product»i vi ty vln school s , 

«... t ^ 

of education as a '"unitary c,once»pt. 

Finally, citation analysis, a reliable process with a 

• ■■ • . ^ '■ 

compelling and Ijjgical basis, produced data w^g,h .showed a 
•stronger relationship th^n did any other measures with funding 
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• 



. 1 ^ 

and* institutional variables. The evidence in this- study iirtplied 
that citation counts are most central to the concept of 
j)rodu,?li vi ty/ Although all measures are somewhat flawed, 
particularly when used to evaluate indivikdual careers, the 
citation rate may' be the besjt single measure "of scholarly work. 




/ 
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